Scheduling Human Intelligence Tasks in Multi-Tenant Crowd-Powered Systems

Micro-task crowdsourcing has become a popular approach to e↵ectively tackle complex data management problems such as data linkage, missing values, or schema matching. However, the backend crowdsourced operators of crowd-powered systems typically yield higher latencies than the machineprocessable operators, this is mainly due to inherent efficiency di↵erences between humans and machines. This problem can be further exacerbated by the lack of workers on the target crowdsourcing platform, or when the workers are shared unequally among a number of competing requesters; including the concurrent users from the same organization who execute crowdsourced queries with di↵erent types, priorities and prices. Under such conditions, a crowd-powered system acts mostly as a proxy to the crowdsourcing platform, and hence it is very di cult to provide e ency guarantees to its end-users. Scheduling is the traditional way of tackling such problems in computer science, by prioritizing access to shared resources. In this paper, we propose a new crowdsourcing system architecture that leverages scheduling algorithms to optimize task execution in a shared resources environment, in this case a crowdsourcing platform. Our study aims at assessing the e ciency of the crowd in settings where multiple types of tasks are run concurrently. We present extensive experimental results comparing i) di↵erent multi-tenant crowdsourcing jobs, including a workload derived from real traces, and ii) di↵erent scheduling techniques tested with real crowd workers. Our experimental results show that task scheduling can be leveraged to achieve fairness and reduce query latency in multi-tenant crowd-powered systems, although with very di↵erent tradeo↵s compared to traditional settings not including human factors.